Connecting to a Windows machine with Ansible

There are two places where Ansible looks for group_vars subdirectory:

the playbook directory - the one containing a playbook you run
the inventory directory - the one specified with -i option at run time, or a default one (usually /etc/ansible, or /usr/local/etc/ansible)

In most typical scenario you might place group_vars along with the playbook.yml.

The example you posted uses the second method, so to reference the files you need to add -i to the command:

ansible windows -m ping -i ./windows


Git & Jenkins: get latest green commit on branch

You don't make any mention of the scripting language you want to use, so I will talk specifically about the HTTP requests to the BitBucket API:

Assumptions

If you have a BitBucket Repository that has three commits in in it the first and the last are failing the build, the middle is passing:

4768815 ❌
49d7110 ✅
42d357f ❌

Get the list of commits

You can get the list of commits by calling the following API method:

  https://api.bitbucket.org/2.0/repositories/{{owner}}/{{repo_slug}}/commits

  owner: RichardSlater
  repo_slug: greencommitproofofconcept

The response looks like this:

{
  "pagelen": 30,
  "values": [
    {
      "hash": "4768815fdc27abf4be17096e7c460f7f68f5d39b",
      "repository": { ... },
      "links": {
        ...
        "statuses": {
          "href": "https://api.bitbucket.org/2.0/repositories/RichardSlater/greencommitproofofconcept/commit/4768815fdc27abf4be17096e7c460f7f68f5d39b/statuses"
        }
      },
      "author": { ... },
      "parents": [ ... ],
      "date": "2017-04-10T11:38:18+00:00",
      "message": "README.md edited online with Bitbucket",
      "type": "commit"
    },
    {
      "hash": "49d7110b98616358d16055960a4abdf2926b890d",
      ...
    },
    {
      "hash": "42d357f1df7a7d7bcf1f10a9f3a5a40d85d5b11c",
      ...
    }
  ]
}

If you parse the JSON and loop over the responses you can extract out the statuses from:

values[n].links.statuses.href

Where n is the index, i.e. 0, 1 or 2 in the above example.  If you were to construct this from scratch it would be in the following format.

Get the list of statuses from the commit

  https://api.bitbucket.org/2.0/repositories/{{owner}}/{{repo_slug}}/commit/{{sha}}/statuses"

  owner: RichardSlater
  repo_slug: greencommitproofofconcept
  sha: 4768815fdc27abf4be17096e7c460f7f68f5d39b

Note: this is a Hypermedia API which means the urls could change so I would recommend using the links from the previous response rather than trying to generate them from scratch.

The response from the above HTTP request will be something like:

{
  "pagelen": 10,
  "values": [
    {
      "key": "POC-01",
      "name": "Build #1",
      "repository": { ... },
      "url": "http://devops.stackexchange.com/q/809/397",
      "links": { ... },
      "refname": null,
      "state": "FAILED",
      "created_on": "2017-04-10T13:04:28.261734+00:00",
      "updated_on": "2017-04-10T13:04:28.261759+00:00",
      "type": "build",
      "description": "Changes by Richard Slater"
    }
  ],
  "page": 1,
  "size": 1
}

From this response you can extract the state using:

values[n].state

Again where n is the status - there could be many of them if one commit resulted in many builds.

If the state for the build you care about is SUCCESSFUL then you have your answer and you can immediately return the sha for the commit.

Loop over all of the commits from the first phase, if you run out of commits follow the next page link that is included in the call to /commits.

Complete Flow Diagram

At a high level the flow will look like this:

Don't forget this is a Hypermedia API so wherever possible have your code follow the links in the API rather than trying to "guess them".


Unable to build Image as it is asking some input

The docker build process is non-interactive. You have to find a way to make it not ask for anything. The way to do that is by creating the necessary config files and placing them in the correct locations through the Dockerfile before installing vsftpd. The questions are there simply for vsftpd to generate a/some configuration file[s] for itself. You create one beforehand and place it correctly using the copy command before run yum install. If vsftpd finds the valid config file[s], it won't ask for anything anymore.


I agree with those who have commented that the role you describe doesn't really match a typical DevOps role. However, there are certain things you won't be able to change in your organization. Instead, focus on the things you can do.
Understand the Objectives
The best place to start is with a clear understanding of Service Level Objectives. What kinds of things do external customers actually care about? If they don't care about uptime, don't make uptime measurements a high priority. Perhaps latency, software updates, content accuracy, or some other measure is more important. Also be aware of your internal customers' needs (your boss, coworkers, etc). What things do they care about?
Once you understand what is most important to your customers, measure, measure, measure! Get as much data as you can about those items that matter. How long does it take to deploy code? When you close a ticket, what percentage of the time does it come back unresolved? And so on. Use those measurements to improve over time.
Here are some areas you can focus your efforts (varies, depending on the company you work for):
Configuration
If someone logged into a server and broke the configuration, how hard would it be to correct the error?

You should be able to consistently build and replace resources with minimal effort.
Configuration management should be complete so that all meaningful changes are tracked.
Ideally, build up processes or tooling as needed so that developers can modify configuration themselves in a controlled and collaborative way (:D DevOps!).

CI/CD
When new code is released, how does it get delivered?

When code is pushed to source control, it should trigger automated testing and deployment. If possible, this should happen in a pipeline that deploys multiple environments in sequence as tests pass (dev -> qa -> uat -> prod).
Improve visibility so that you can easily see where different versions of the code are running, what errors are occurring, and how performance is changing.
Enable your software developers to deliver code themselves (:D DevOps!).
Collaborate with developers to improve this process (:D DevOps!)

Logging/Metrics
When something goes wrong, how quickly can you find out what the problem was? If something went wrong 2 months ago, can you still find out what the problem was?

Collect logs in a central place with tools that help you search and visualize.
Aggregate metrics with tools that help visualize.
Reduce noise! If it's not meaningful, don't keep it (but be a little conservative, sometimes things turn out to be meaningful later).
Work with developers to understand how logging and metrics would help them deliver better software (:D DevOps!).

Alerting/Incident Response
Ideally, you should not ever need to look at a graph, dashboard, or log message, unless there's an actual problem to solve. Sitting and watching a dashboard is not a good task for humans.

Set up alerting so that you get notified when problems arise.
Alerts should go to ONE person, who is the RIGHT person.
Alerts should be urgent (needs to be addressed immediately), and actionable (the person getting the alert can fix it).
In the spirit of paging the right person, you may be able to work with developers on getting them into some kind of on-call rotation (even just M-F 8-5 for a subset of code issues). (DevOps?)

Disaster Recovery
If your servers were destroyed today, how long would it take to be up and running again?

Take backups of data and set up backup infrastructure.
Test backup recovery regularly.
Test failover to secondary infrastructure regularly.
Work with developers to better understand how code will operate in a failover scenario and improve as needed (:D DevOps!).

Documentation
Heaven forbid something happen to you, but if it does, how well can someone else replace you. Document what you've built and improved so clearly that another person can replace you. Don't worry, this makes you less likely to be fired.
Education
Always, educate yourself. Take online courses, read books, work on certifications. I don't know what the path looks like for you. I believe that spending a few hours every week specifically dedicated to improving your own skills is a valuable use of your time (even for your employer!).
A book like Site Reliability Engineering (Beyer, Jones, Petoff & Murphy) or The Phoenix Project (Kim, Behr & Spafford) might be good to read on this topic.

Totally possible I've entirely missed the point of your question. If so, I apologize. However, I think this ought to give you some ideas of how you can spend your time in ways that is worthwhile to both your employer and yourself.


Is there a common name for describing "spans" of versions?

I have never heard of a common name for what you describe; and rarely, if ever, have I seen a notation where someone references to something which has a "begin" and "end" version. It just is not that applicable in a general fashion; this is probably why there is no such name. Also changelogs usually don't contain this information except for extremely bad breaks that need some kind of notice about which span(sic) of versions one should rather stay clear of.

The closest you could get to a common name (popularized by Jira, for example) is probably to call 1.3 in your example the "affected version", and 1.5 the "fixed version"; possibly allowing multiple values there. But they also have no name for the span as an entity.

Note that it is not really a problem if you have multiple old maintenance releases like you are describing. Those would just lead to multiple spans distributed over the branches.

Regarding your edit:

  Instead of having to look through release notes this would be a table of features mapped to versions. Release notes typically have "bugs not fixed" but release notes never show bugs not yet found. My customer has multiple variants/branches of systems deployed all over the world, and wants a quick and easy way to know which variants and deployments have which features and bugs.

Been there done that. You are doing well to approach this with care, it can really "explode" after a while. If you are using a tool like Jira, you should really find a field for each issue which allows you to mention releases related to the issue. If you decide do go manually, just list your version ranges in whatever format seems right to you.

I would not bother with finding a name for a range of versions, but just call them "affected versions".


Jenkins downstream pipeline jobs not always triggered as expected

I suspect this job name is the issue:
build job: 'second_job_name'

In a Multibranch pipeline, there are typically folders in the job name.  For instance, if you have a Multibranch project named "my_project", a repo in that project named "my_repo", and a branch within that repo that you wish to build, let's say "master", then your build command would look like:
build job: 'my_project/my_repo/master'

If you only specify up to the project or repo level in your job name, but don't include the branch, instead of running a build for the branch, Jenkins will scan for updates to all branches within that project/repo, and build any new commits which it hasn't yet built.  This results in exactly the symptoms you described:

The only time this setup works as expected, is when there are new commits to all 3 git repos. Then all 3 jobs are triggered sequentially without issues.


How to get list of env variables from the root folder to jenkins job?

I recommend that you use the printenv command for list system Variable:

printenv

and also An easy way to obtain the Jenkins environment variables list from your local installation is to append env-vars.html to the server's URL. the URL would be

http://{jenkins_url}/env-vars.html


Testing docker image builds when low level libraries change

So I believe I came up with a somewhat re-usable solution for this problem using devpi. It allows you to use the same Dockerfile for the testing and production image.
Devpi allows me to create my own index which can shadow my production index with versions of software that haven't been officially released.
Steps to make this work:
1. Base image from ARG
I first edited my Dockerfile to allow overwriting the base image used. This will allow me to create a layer in the middle that configures pip to talk to devpi:
# Builds image reistry.myorg.com/python-app
ARG BASE=reistry.myorg.com/python-base:latest
FROM $BASE

COPY . /app

# Implicitly gets the latest `myorg.lib1` and `myorg.lib2` from PyPI
RUN pip install /app

2. Upload test versions of library
For each python change:

Clone the repo
python -m build the repo
devpi upload the wheels

3. Create a layer with devpi pip conf
Make a pip conf that can access your devpi instance and configure to your index name:
[global]
index-url = http://<some ip>:<some port>/root/dev/+simple/
trusted-host = <some ip>
disable-pip-version-check = true

[search]
index = http://<some ip>:<some port>/root/dev/

And add that to your base images pip conf:
FROM reistry.myorg.com/python-base:latest

COPY pip.conf /etc/pip.conf

4. Chain the images together
docker build -t devpi-base -f devpi.Dockerfile .
docker build -t my-app -f Dockerfile --build-arg=BASE=devpi-base .
docker tag my-app reistry.myorg.com/my-app:test1
docker push reistry.myorg.com/my-app:test1

Now when pip installs all of your dependencies you can have it pick up lower level changes to python packages. This concept should be re-usable for a lot of languages I imagine!


Software Security Testing in a DevSecOps Pipeline

I agree that relying on "developer education" is quite futile, at least when it's the sole course of action. Combine this, however, with automation and you can achieve far greater impact. Both sides, DAST and SAST can be covered with:

Unit test authentication: Make sure that you have unit tests that check registration, login, password reset and that it is integrated with your development process (continuous integration). Implement processes that automatically block changes to your code when they fail.
Testing against known vulnerabilities: These are automated DAST tests that test your application from outside. Programs that handle this task include BDD and mittn which is inspired by GAUNTLT.

Integrate everything properly with a CI server (Jenkins, Gitlab, Travis) and stop having to rely on your programmers' "best effort security"

Note: my opinion of proper CI integration, from a language/platform agnostic view, covers the following

Tests/examples:
 - unit tests that cover at minimum 80%, better 90%
 - integration tests that ensure the modules work with one another
 - example programs that show how a user should use the API

Build automation:
 - Automatically build every pushed commit
 - Build library, test and example code
 - Build for each platform/configuration
 - Run the tests

Merge/pull requests:
 - Block merges into master branch if any thing fails
 - Require code review of at least one other colleague

Code analysis:
 - Run nightly static code analysis (style, anti-conventions, dangerous constructs)
 - Run nightly dynamic code analysis (memory leaks, race conditions, known security flaws)

Publish results:
 - Publish graphs/trends of test/analysis history
 - Talk about changes/targets in weekly meetings
 - Act on problem areas with sprints/work packages


There is no default password, actually none at all. You must create a new user with

➜ npm adduser --registry  http://localhost:4873/
Username: wow
Password:
Email: (this IS public) verdaccio.does.ignore.email@whatever.com
Logged in as wow on http://localhost:4873/

Then you are logged in.


How do I implement user authentication and management in Kubernetes?

I'm trying to architect a micro-service application, that runs in Kubernetes. It consists of several RESTful APIs. I want a central point to implement authentication for these APIs.

This is a good idea.

So what I'm looking for is a/the way to have central authentication in Kubernetes for users accessing the APIs. I would also require to be able to programmatically add users to this auth layer, so I can have users sign themselves up for access.

Just to make it clear: Kubernetes is a container orchestrator, e.g. a way to manage your services. It consists of a control plane (think management of apps) and a data plane (think your apps and customer traffic). Both parts may require proper authentication and authorization. When it comes to your "business idea" - it is about the data plane - here this problem is orthogonal to the platform (e.g. Kubernetes or Heroku) that you run your apps on. So think how you should solve your problem with Microservice architecture, and not so much about Kubernetes.

In my product, users would be able to sign themselves up on a website. Signing up would have to give them access to the API's.

The most common protocol for this nowadays is to use OpenID Connect - that is the protocol behind both e.g. Google Accounts and Microsoft Azure AD. What you need is an OpenID Connect provider implementation. The most commonly used open source implementation is Keycloak. Remember, this kind of service is stateful and you will need a database as well.
Instead of setting up your own OpenID Connect provider, you could also let the user use an account that he already has, e.g. a Google Account or Twitter Account - it depends on your use case.

I've seen ingress controllers that do central auth, but I haven't found if they support programmatically adding accounts. I've also seen API gateway software that might do what I want, but I'm not sure if and if so, which ones can run in Kubernetes.

What you talk about here is mostly validation of authentication and this is sometimes done using "auth aware ingress controllers". With OpenID Connect protocol, the user send a JWT-token as proof of authentication, and you could redirect unauthenticated users to your Open ID Connect provider.


Where does TFS 2017 actually store artifacts?

There are two options when publishing artifacts: Server and File Share.

If you publish to the server, they are stored in the TFS database. You can't configure the location, however you can configure retention policies to ensure that old builds aren't kept around forever.

If you publish to a file share, they're stored on, well, a file share.


Get all change logs of since last successful build in Jenkins Pipeline

You may try something like this:

node('my_kubernetes_pod') {
    passedBuilds = []

    lastSuccessfulBuild(passedBuilds, currentBuild);

    def changeLog = getChangeLog(passedBuilds)
    echo "changeLog ${changeLog}"
}

def lastSuccessfulBuild(passedBuilds, build) {
    if ((build != null) && (build.result != 'SUCCESS')) {
        passedBuilds.add(build)
        lastSuccessfulBuild(passedBuilds, build.getPreviousBuild())
    }
}

@NonCPS
def getChangeLog(passedBuilds) {
    def log = ""
    for (int x = 0; x < passedBuilds.size(); x++) {
        def currentBuild = passedBuilds[x];
        def changeLogSets = currentBuild.rawBuild.changeSets
        for (int i = 0; i < changeLogSets.size(); i++) {
            def entries = changeLogSets[i].items
            for (int j = 0; j < entries.length; j++) {
                def entry = entries[j]
                log += "* ${entry.msg} by ${entry.author} \n"
            }
        }
    }
    return log;
}

Additionally, you may try out "Changes Since Last Success Plugin":
https://wiki.jenkins.io/display/JENKINS/Changes+Since+Last+Success+Plugin

But it will not work together with pipeline, this is a separate approach.


AWS-CLI : to download IAM user with policy

You can use the below command to get all the policy attached to an iam user. (You must be having permissions to list-users and list-user-policies before using the below commands)

aws iam list-attached-user-policies --user-name username | jq -r '.AttachedPolicies[].PolicyArn' | awk -F'policy/' {'print $NF'}

(Make sure to replace username with some actual iam username in the above command)

If you want to get the list of all users and user-policies attached to those users, you can use the below one line command. The below command will get all the iam users and then will find the user-policies attached to that user and will put in a CSV file. The first column will contain username and second column will contain `policies separated by spaces.

for u_name in $(aws iam list-users | jq -r '.Users[].UserName') ; do permissions="$(aws iam list-attached-user-policies --user-name ${u_name} | jq -r '.AttachedPolicies[].PolicyArn' | awk -F'policy/' {'print $NF'} | tr '\n' ' ')" ; echo "${u_name}, ${permissions}" >> iam_users.csv ; done


Is it possible to view all changes made on a Git feature branch in Azure Devops?

You're looking for the branch compare functionality.  In the list of branches, click the overflow ellipses on your feature branch and select compare branches.

Azure DevOps will diff you branch with the default branch which is changeable if necessary.


Start only one pod at a time in kubernetes

Use a combination of Readiness Probes and a proper deployment strategy.

Configure your Deployment spec like so:
spec:
  replicas: 4    # or as many as you need to have
  strategy:
    rollingUpdate:
  maxUnavailable: 0
  maxSurge: 1

maxUnavailable = 0 means that there should always be a number of pods available equal to the replicas count. maxSurge = 1 means that kubernetes will create the new pods one at a time, and destroy the old ones one-at-a-time as well.
Implement a readinessProbe which should return a successful response when the database migration has completed. You'd have to write the logic for that yourself, but if you do it right, kubernetes will create a pod with the new Deployment and wait for it to complete and only after that, evict 1 of the old ones.

PS: It's worth noting that this approach will cause your node to temporarily contain more pods than your replicas config implies, namely replicas + 1 for a short period of time while deploying. You need to make sure that your node can handle the little extra load untill the rollout is complete.


Jenkins cron works only for the 1st line

I believe your issue is the use of * * * * *.
The parameterized-scheduler-plugin example has two different times specified.
#lets run against the integration environment at 15 past the hour
15 * * * * % env=int
#run QA too
30 * * * % env=qa

So one job runs @15 and one @30.
Yours specify the same time, so only the first is executed.
You could try the random time parameter "H" or fix some different interval on each.
Or, it does say "Leave a space. Put in a %. Then add the name=value pairs". So maybe it's cronexp space percent space name=value ?
OR you could just copy the job job and schedule two separate jobs w/fixed parameter.
But Jenkins does sound like the wrong tool for this.
The above will address why only the 1st line was executed.
NOTE :There is a known issue attemptimg to to every minute, see JENKINS-22129 H/2 will run every 2 minutes, but H/1 will run every hour.


Why can this ENV not be found when it is defined in one layer using a Dockerfile?

The ENV keyword will be translated to export command (builin of the shell usually), if you remove the multi-line declaration you end up with this equivalent:

export A=123 B=$A

What happens here is that when the shell parse the line to give it to export input, A is not yet exported and available as an environment variable.

Step by step this will give:

You type export A=123 B=$A
The shell parse the line and pass A=123 B= to export's stdin
export parse each argument in the form key=value and as such export A to be 123 and B to be empty

On the second form, A is available to the shell when the line is parsed, and as such the resulting export's stdin is B=123


How to push to other origins using Azure Pipelines?

While I haven't tried this, I suppose a solution would be to have a regular pipeline (this should work in either build or release pipelines) that would check out your repo (as it's default first step) then some scripted command that adds the desired remote and pushes to it.
Some pitfalls I could see you running into:

issues with the state of the agent work folder based repo prior to a push; but that should be reconcilable with the appropriate Git commands. If the target remotes (i.e. backups) are never being committed to by anything other than the pipeline, you hopefully won't run into conflicts
arbitrary branches: I'm not a git expert, but I think there's a way to push all branches to the remote
authentication: you'll obviously need to provide appropriate authentication to the target remote(s). Credentials shouldn't be stored in the scripts in plain text. Use secured pipeline (or group based) variables
repo size: a huge repo will need to be cloned into the ADO agent workspace. Not a problem as long as you have the disk to support it.

Given the needs you listed, it seems like you should consider something like AzureDevOps services in the cloud, or private GitHub repos (recently became free), etc. Why manage your own backups if the cloud can do it for you?


Docker Containers fails to DNS resolve on subnet 10.0.2.0/24

We have not been able to figure out what the problem is or what a solution could be.

The workaround is actually not using the 10.0.2.0 subnet, so I will add that as an answer as requested in the comments.

If someone finds the real issue and solution here, I will gladly accept that answer though.


Docker tags for multi-stage builds for series of Docker images out of one Dockerfile

You can build and tag any stage of a multi-stage build. Just use the --target option to docker build. E.g.

docker build --target jdk -t myapp-jdk:v1 .
docker build --target jre -t myapp:v1 .

With the layer caching, the the earlier steps will be reused from the cache and not rerun from scratch each time, so there's little downside to running the build to different targets multiple times.

If you absolutely need to run this from a single build command, you can parse the output of the build, get the image id of the intermediate step, and tag that image id:

docker tag <some_image_id> myapp-jdk:v1


It has been a while since I have run a few Wordpress based sites. The database stores some of the content of the site. The file system stores media that is uploaded into the site. More significantly Wordpress is a Content Management System. The intention is that your site users can add content and media to the live site. When someone uses the WYSIWYG editor to add or edit the content you will find that the sites domain name leaks into the database. So if you move the Staging database into Live you would get links appearing on the Live that that point back to the Staging site.
I quick bit of research to see if things have improved since I last used it leads to this article on putting Wordpress into git. Please read the first article in that series that explains the challenges. You will need to understand a little of how Wordpress works to understand the recommendations I make below.
That article talks about turning off automatic updates so that you have to manually security path. I am pretty sure that I remember that updates can write things into the database. I would be very worried about turning off updates as there are a huge amount of attacks against Wordpress as it is so widely deployed. You really should avoid deploying many third party plugins as they might not get security patches in time. This isn't a theoretical problem I had quite a time dealing with Wordpress being used as a spam relay. Yet even if you follow that article you won't have solved the database migrations issue.
My recommendation is that you take a step back and try to figure out what problems you are trying to solve and how to adapt the tools and process:

What is the developer developing? A custom Theme and/or Plugin? If so then this article covers that. Put the Theme/Plugin into its own repo.
Who actually writes the site content or uploads media? The general public or members of staff? If it is staff then maybe they need minimal training in a Test environment and can happily edit Live after that.

Given that Wordpress does not let you aim for perfection due to the way that it is built, yet it is hugely successful, carefully pick which of the following you are aiming to optimise:

A Specific Service Level Agreement
Meantime to recovery
Mean time between failures
Reducing the commercial impact of errors
Something else?

There are quite a few other specifics that may affect what set of compromises might work best for your situation.
Based on my experiences without actual knowledge of your specifics:

Give up the idea of promoting changes through Dev, Staging and Live. I would simply have a Test site and a Live site. I would periodically refresh the Test site from the Live site. To do this I would dump out the entire Live database a run a sed -i over it to replace every instance of "www.mysite.com" with "test.mysite.com".
Give up on using Docker as the software distribution mechanism for the actual site. I would use Docker to supply the PHP+Apache runtime and mount a volume that contained the actual site php+content. This because the disk is a combination of the code and the content of the site so you need to be able to backup and restore it independently of changing the Docker image.
Try "git checkout vX.Y" to update the live site in a persistent volume mounted into your generic PHP+Apache docker image. I would investigate the ideas in this series of articles.
Automate the backup of the media content volume every night and keep as many backups as you can. This is why it needs to be a volume mounted into Docker.
Automate the backup of the database every night and keep as many as you can.
Automate the restore of the site as a script that restores the database to the last midnight backup then restores the last nightly snapshot of the media content folder.
Avoid any party plugins that are not commercially supported. They likely won't work at the next major upgrade of Wordpress and will pin you onto a version that will become obsolete.

Then they workflow that I would go for would be the to add "experimental features" into the Test site before applying them to the Live site. This would be based on git checkouts in the best case. Yet I would also consider doing Wordpress security updates via the CLI. Those can update the database and the state on disk underneith Docker hence I do not recommend you try to mange the state of your site as a Docker image.
You then have to worry about the Test site drifting away form the Live site. So I would on a periodic base (at least monthly) overwrite the Test site with the Live site. This would involve:

Backing up the Lite sites media content on the persistent volume and restoring it over the Test sites media content on its persistent volume.
Backing up the Lite sites database, doing a sed -i to rewrite any content links pointing at the Live to point at the Test site. Then restoring it over the Test database.

A key part of this approach is that you would be running your Live site like every other Wordpress site in the world. You don't do "brain surgery" on the Live site. You do frequently do "brain surgery" on the Test site. If the brain surgery goes wrong it's annoying and painful to fix but it is only the Test site that needs some new step to bring it into sync with Live. It is likely that this only happens when you move to a new Wordpress version. If you try a custom approach of moving change into a Live site they might break at a future Wordpress release and leave you with a multiple-day outage in Live.
I completely recognise that the approach outlined above isn't at all how "normal" software is developed. Yet it is the approach that I have scripted in the past. Even if you did the standard dev/test/live approach you do not guarantee 100% perfection and 0 bugs but you might have a longer mean time between failures. If you accept that perfection isn't possible then what you want is very fast rollback to reduce the impact of bugs. You do this by focusing on reducing the meantime to recovery to the speed of a restore script and the time taken to get permission/agreement to run it.
Being able to rehearse things in a Test environment should mostly avoid a total outage on the live site. Being able to do a fully automated restore can then bring you back to a good state at the loss of some recent edits to the site. That is normally completely acceptable compared to having an extended outage. Consider this scenario:

Staff are continually editing content in production including uploading new media into the content folder on disk (e.g., uploading a PDF and linking to it within content where the link is in the database).
You have run some Wordpress updates in Test a day ago and they seemed good. Today you run them at 9 am in the Live and something odd happens (maybe a timeout, whatever) and the site becomes corrupted.
You have automated the rollback. You run the roll-back script which computes the last nightly full backup point then restores the database, then restores last night's media content folder backup over the live folder to revert any lost or corrupted files.

To my knowledge that is as good as you can get. Your meantime to recovery should be the speed of your database restore plus the time take to spot the problem and agree to do the restore. This is exactly why you shouldn't have to speak to editing Staff about whether or not they are happy with the roll-back as the site is broken so of course, they would agree to restore the site automatically, rather than have the site down for hours while a possible point fix is investigated and attempted. You can send out a prepared email telling Staff that there was an outage at 9 am such that the site has been restored to the previous nights backup and that they should double-check and repeat any work that they have undertaken in the past twenty-four hours and redo their work as necessary. IMHO that is as close to perfection as an opensource PHP content management system such as Wordpress allows.


packge.json not found while executing on Jenkins

Because you are setting the container's WORKDIR to /app/build so your package.json is not in the working directory, but in the parent of the working directory.


Deploying and testing features on a server by more than one developer

Since you have a single released version product one thing to consider would be keeping the Develop branch or not:

keeping it would have the advantage that at any moment you can bring in a hotfix in master even if Develop isn't stable enough for a merge into master. But this means you'd need to have 2 focus points instead of a single one - both branches need to be kept stable. There will be additional efforts for constant merges back and forth between the 2 branches to keep them from diverging. The overhead of context switching between them can be significant.
getting rid of it (see How to get rid of develop branch for simplified Git flow) and doing the integration directly in master would reduce the overhead of carrying together 2 branches. The releases would simply be tagged versions of the master branch. But getting in a hotfix for the current release version would require the branch to be stable enough (virtually ready for a new release) at any moment. Which would be desired anyways and is achievable (with some effort).

In the context of continuing the practice of developing features in isolation (on separate, potentially long-lived feature branches) and going through the pains of integration hell one thing that could be improved would be a better communication between the business owner and the development team: deciding to drop feature B after it was integrated into the product is practically equivalent with adding a new feature C (late in the game).

Another (much better IMHO) option would be to switch to the Continuous Integration practice: features would be developed directly in master (using very short lived branches carrying tiny incremental changes much easier to handle, ideally merged back into master in less than 1 day), behind feature toggles/feature flags:

during development the features would be disabled by default, making master virtually equivalent to the current release version, ready to receive hot fixes if/when needed.
the flags can be individually enabled in various environments to determine the respective feature progress/readiness for release. This can be done simultaneously for virtually any number of features (or feature combinations), regardless of their size or development stage or duration, as different verification steps of the automated CI system.
whenever a feature is ready and the business owners decide to accept it the respective flag is turned on in a tiny commit and a new tag "brings" the feature into the next release. No massive branch merges, no risk of surprises, everything should be just business as usual - all kinks should have been already ironed out. A much more predictable (and less stressful) development process.
keeping the flag supporting code in place for a while also allows staging the dropping of the feature in a subsequent release by disabling the flag in a CI system verification stage. But eventually a cleanup of the feature flag and the supporting code can/should be done as a background code base maintenance task, making the feature permanent.

IMHO such approach is the only reasonable option in larger scale projects where many features are being developed simultaneously.


Saltstack howto create a Concatenation CA?

It feels like what you have done here is implement a discount/bastardized CA (Certificate Authority). I fear that in doing so, you might have inadvertently created a security threat for yourself.
In a traditional PKI deployment, your client would generate a private key and a Certificate Signing Request (CSR) that is then submitted to your Certificate Authority for signing. Your CA would then sign your key after validating the applicant by some means and issue a Certificate to be installed on the client.
What you may have just done by concatenating every single certificate of each client into a CA is grant each and every client the ability to sign certificates for other machines. Any machine in your Gluster cluster then may have the ability to issue a certificate for an arbitrary key which then would be trusted by your Gluster cluster which could allow for an attacker to use a compromised system or key to allow an arbitrary endpoint to join your cluster.
Instead, you might want to follow this guide on setting up a CA for use by your SaltStack. In fact, it is worth noting that your salt master is itself a CA. You could simply copy the minion's certificate and key for use by gluster, though that has the downside of a compromised cert/key being able to be used to impersonate a minion.
But generally speaking, you probably don't want to grant all of your minions the power to be a CA.


Can a server used for GitLab runners be used as a deployment server?

GitLab and GitLab CI are not two servers: they are softwares installed on a server. Now, while both can be installed on the same server, they can be split for scaling reason.

Now your question: it all depends on what type of services your are running. Bear in mind that CI can take CPU/memory and disk bandwidth while running your CI (docker image build, composer, npm, Java, etc.). Doing so can slow down your production server.
Another reason not to run docker-compose directly is to avoid downtime: what if there is a mistake during the build or the deployment? Your production will be down. You may want to look at blue/green deployment.


How to get traefik ingress to load balance across several ClusterIP services

The sample I posted initially works as is:
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: my-service-ingress
spec:
  entryPoints:
    - web
    - websecure
  routes:
    - kind: Rule
      match: Host(`myservice.mycompany.example`)
      services:
        - kind: Service
          name: myservice-node1-clusterip
          passHostHeader: true
          port: 80
        - kind: Service
          name: myservice-node2-clusterip
          passHostHeader: true
          port: 80
        - kind: Service
          name: myservice-node3-clusterip
          passHostHeader: true
          port: 80
        - kind: Service
          name: myservice-node4-clusterip
          passHostHeader: true
          port: 80


passing args to shell module in Ansible adhoc command

Append the parameters after the command. See the example below.

# ansible -m shell --args "pwd; touch test; ls -1 chdir=/scratch/tmp creates=/scratch/tmp/test" localhost
localhost | CHANGED | rc=0 >>
/scratch/tmp
test

# ansible -m shell --args "pwd; touch test; ls -1 chdir=/scratch/tmp creates=/scratch/tmp/test" localhost
localhost | SUCCESS | rc=0 >>
skipped, since /scratch/tmp/test exists


Build isolation - Docker container based builds in Bamboo 6.4.0

Yes, you are correct. Bamboo will spawn the build containers on the agent that the build is running on (either Local, Remote, or, Elastic - depending on your configuration).

In order for this to work the agent will require Docker to be installed and running.


How does jenkins publish over ssh store passphrases?

According to the source code for the Publish Over SSH plugin, the passphrase is encrypted and decrypted using hudson.util.Secret, and the encrypted value is stored in the plugin's XML configuration file.

Supposedly there is ongoing work to switch to using Jenkins' built-in credentials store; however, I don't see any active pull requests for such a change.


When I lookup your domain thuijls.net on MXToolBox I get the following messageL

Dns record not found

They explain this to mean:

  More Information About Dns Record Published
  We did not find a SPF (TXT) record for the domain. Either you are using the deprecated type "SPF" record (in which case you will see a warning message for this alongside this message) or no record exists.

  For a SPF record to be valid, it must at least contain v=spf1

A TXT lookup for thuijls.net yeilds:

Type   Domain Name     TTL     Record
TXT    thuijls.net     5 min   v=spf1 include:spf.messagingengine.com ?all

The problem is the ?all - you probably copy-pasted this from somewhere and the (non-standard) hyphen became a question mark.

Edit your TX record to read v=spf1 include:spf.messagingengine.com -all and all will be good.

Then you can add a 2nd TXT entry for your  "protonmail-verification-da....." setting.


Ansible - stop when host is success, exit with IP

The thing that makes this complicated is that ansible registers variables and facts locally, on the target remote host - and then has a different set of variables and facts once it moves to the next host. So it would require a bit of fiddling to do this exactly the way you want.

One way to approach this would be to just use run_once, but it doesn't work with serial: 1 which you are asking for (check one host at a time);

---
  - name: test
    hosts: all
    vars:
      mem_to_look_for: 62950
    tasks:
      - name: do things
        debug:
          msg: "hello"
        run_once: true
        when: ansible_memory_mb.real.total==mem_to_look_for

If you really want it to run serially, we could borrow some of the knowledge out of this article https://stackoverflow.com/questions/42376123/how-to-terminate-an-ansible-playbook-run-when-it-is-successful-on-one-host and tweak it a little.. and it works great!

---
  - name: "test"
    hosts: all
    gather_facts: no
    serial: 1
    vars:
      mem_to_look_for: 62950
    tasks:
      - meta: end_play
        when: "{% for item in hostvars.values() %}{% if item.done|default(False) %}True{% endif %}{% endfor %}"

      - name: gather facts
        setup:

      - name: ensure host has the memory we're looking for
        set_fact:
          done: true
        when: ansible_memory_mb.real.total==mem_to_look_for

      - name: skip this host if it doesn't have the memory we're looking for
        meta: end_play
        when: ansible_memory_mb.real.total!=mem_to_look_for

      - name: run this thing only on one host
        debug:
          msg: "running the thing"

I set the playbook to gather_facts: no because we don't want to continue gathering facts once we found the host we're looking for. It doesn't abort the playbook unfortunately, but just kind of runs a no-op on the remainder of hosts. You could also probably speed things up by tweaking the setup: module by specifying which subset of facts you want to collect, but I left it to collect all the facts in this example. https://docs.ansible.com/ansible/latest/modules/setup_module.html you may also want to just move the host selection to the setup module by adding failed_when: ansible_memory_mb.real.total!=mem_to_look_for at the end of the setup module, but then the hosts would be marked as failed. I'm not sure if that's what you're looking for, but it's really up to you on if you want them to be failed or just skipped.

Additionally, if you want it to completely stop checking hosts as soon as its done I think the any_errors_fatal property will do that. https://docs.ansible.com/ansible/latest/user_guide/playbooks_error_handling.html#aborting-the-play


What are the measures of size and complexity of IT infrastructure

TL;DR: Measures of size can be split roughly in three different categories that I would define as depth in what you
  manage vs. what you outsource or consume as a service, breadth of
  services supported and height in numbers of instances,servers and
  customers. Measures of complexity largely depend on selected
  systems architecture, organizational structure of people supporting it and skill sets required. Where depth and skill sets
  go hand in hand adding both to size and complexity.

Size mesures

Note: Most of the following size requirements can also add to complexity through systems architecture, skill sets and organization structure needs.

Depth of Infrastructure

How much of the infrastructure you are outsourcing to others in increased levels of depth:

Do you simple use Software as a Service for all that you do?
Do you operate fully in public cloud, private or hybrid cloud or
using some PaaS?
Do you use Infrastructure as a Service?
Do you use Hosted and Managed Infrastructure in rented DC space?

Do you own or rent the hardware?
Does the provider manage infrastructure monitoring?
Does the provider manage basic system administration?
Does the provider manage hardware failures and maintenance?
Does the provider manage racking and server installation?
Does the provider manage internal networking?
Does the provider manage internet connectivity and routing?

Do you have just data centers with contracted remote hands?
Do you host everything on premises or in your own datacenters?

Breadth of Infrastructure

What are the different types of services you support?

Compute Resources

Bare metal servers
Virtualization layer (VMWare)
Container layer (docker, k8, mesos)
Serverless layer (lambda, functions)

Storage Resources

Standalone storage appliances
RAID in servers
Large standalone relational database clusters
Time series databases
Object store clusters

Network Resources
Observability Resources

System Log servers
Metrics and Graph systems
Search clusters

Automation Resources
Backup/Recovery Resources
Complex compounded services

ELK, Hadoop, etc.

Height of Infrastructure

What is the scale for each resources that you need? Do you operate the services on single server/instance or do you need use clusters of machines?
What is the level of redundancy you need?
What are your availability requirements?
What are your requirements for latency and throughput of services?
Do you need geographically distributed infrastructure? (international business, latency requirements or regulation compliance like GDPR, data localization laws, etc)
Do you need multiple datacenters in each geography?

Complexity Measures

Just very briefly...

System Architecture

When it comes to Infrastructure complexity, it quite closely follows the complexity of the distributed systems supported by the infrastructure. You have to take into account two types of systems:

Distributed systems supporting individual services.
Distributed system created by interdependencies of the services.

Distributed Systems Complexity

Every service your infrastructure supports can have in and of itself a different level of complexity with varying levels of requirements on the infrastructure. Systems supporting services can range through:

Single threaded.
Multi-threaded (shared memory, shared disk)
Parallel systems with data sharding
HA Failover (Primary / Standby) (Cold, Warm, Hot)
HA Cluster (N+M)
Real-time Clusters

Interdependence of Services

Let me start with an example. Let's say your infrastructure reports test results into ElasticSearch cluster. Your pager depends on monitoring and test data provided by ElasticSearch. Geographical distribution of ElasticSearch cluster makes it depend on your datacenter networking connectivity. Now one of your internet providers decided to make an unannounced maintenance on Saturday night, throughput drops, your traffic is rerouted to backup provider, monitoring traffic is de-prioritized to customer data traffic, ingest of monitoring events slows and your pager goes crazy.

Every time two services, two parts of the infrastructure depend on each other, they create a new single distributed system, whose complexity should be judged independently. Such dependency can be either removed or reduced. Remember that system is only as redundant and as available as the intersection of all services that it depends on.

Further examples of complexity increasing factors:

Dependencies on external services.
Attempts to mitigate failure from service dependency.

Multiple providers
Caching of data

Organizational structure

This is a chapter in and of itself ... people are often overlooked part of the overall system of IT infrastructure. We rarely think about redundancy, availability and latency factors when it comes to people, but just as with computers, these same issues affect organizations maintaining infrastructure and the complexity of it can sometimes easily outweigh the complexity of the computer systems. People involved in maintaining your infrastructure can span multiple time zones, languages, geographic locations, companies, pay scales and legal codes. Any of those factors are signs of increase in complexity.


Original Helm chart gone - how can I find / get it from the cluster?

You can use helm get manifest <RELEASE_NAME> to fetch the kubernets manifests and you can use helm get values <RELEASE_NAME> to get the values.yaml from the cluster. But I don't think there is any way to get the chart files back. Sorry.

PS: Please replace <RELEASE_NAME> with your release name.


How to incorporate containers into a standardized dev environment?

Docker is an awesome tool for creating a database, seeding data into it, and disposing of it.

For an example check out Microsoft's SQL Server with Docker.  The following would pull and run a sql server 2017 instance: docker run -e "ACCEPT_EULA=Y" -e "SA_PASSWORD="
   -p 1433:1433 --name sql1
   -d mcr.microsoft.com/mssql/server:2017-latestAfter running this you now have a developer ready SQL instance that you can then connect to SQL Management Studio (or any other GUI management tool), or you can even connect via the command line.

If you are not using SQL you can run a container for the desired database type (check out the options on Dockerhub).

Once you have your database up and running, run the scripts to set it up, seed the data, and test without worrying about messing up your production database.


Is it better to perform delayed jobs using queues or at app level?

This sounds on the fence with opinion based but I think there's an important thing that narrow enough the problem to be answered.

It is still an opinion but, as far as you're horizontally scaling, instances may appear and disappear at will, keeping the delay in memory is thus not an option, so at this point you'll have to workaround this and bake something like a notifications queue/table to keep what is to be sent at which time and periodically check what is to be sent.

Once that is said, you'll need a central place somewhere to keep the needed notifications out of the app instance themselves and as such you'll need a place to store them, it could be a redis service, a database table or whatever, you'll be coding it anyway.

When working on AWS, the services provided are already redundant and on high availability without much effort needed on your part, as such I'd go with an application code storing the notification somewhere with the time to send it and a lambda checking periodically which notification has to be sent, the notification can then be sent through SQS to take advantage of the retries in case of delivery failure or you'll have to handle it in the notification firing system.

The store part being elastic cache or something else is a matter of taste and team knowledge in my opinion, take whatever you're more comfortable with.
For the part firing the notifications I'd use something separated from the app itself, another app or a lambda checking periodically the 'cache' (DB or redis or whatever) for notifications to be sent.

This allows the app to be sure the notification once declared and acknowledged by the cache is properly registered and will be fired when needed and retried in case of problem. A last case is the impossibility to deliver the notification and you may have to keep track of this in the application to avoid sending too much notifications ending as dead letter but that's sliding out of scope for this question.


Commit messages from Submodules to Jenkins Change Log

After consulting this cheatsheet, I think that you are looking for the subject. If I apply:

git log --pretty="%s"

to one of my git repositories, the following is returned:

learning pointers in golang
receiver struct in golang
embedded structs in golang
multiple spanish words

In conclusion, I think you should include %s to get the git messages or basically the commit subjects.


What are best practices of deploying docker-compose using Ansible?

Instead of using a symlink that will be stored as is in BitBucket one could use:

- name: Copy docker-compose.yml
  copy:
    src: ../../../../docker-compose.yml
    dest: /path/to/docker-compose.yml


GKE - GPU nvidia - cuda drivers dont work

Ok so i finally made it work.

It is mandatory set a ressource limit to access the nvidia driver, which is weird considering either way my pod was on the right node with the nvidia drivers installed..


How to deploy Node app via Jenkins to Ubuntu(18.04)?

This article very clearly explains how to deploy a nodejs app using Jenkins:

https://blog.couchbase.com/create-continuous-deployment-pipeline-nodejs-jenkins/


Start dev with purely docker, or something like vagrant (in a to-be-dockerized application)?

Vagrant, in this case, would simply add a lot of unnecessary overhead into your testing environment. You can create containers that quickly test specific groups of functionalities and paste the results somewhere. Your team won't spend much time learning how to start these containers as it is usually a simply command. Most of changes are going to be done inside the code, not into the testing environment. But if you need to change the test environment, the Dockerfiles must be in an easy to access place. You could also automate the build process of containers linking the Dockerfile (e.g. from a GitHub repository) into the DockerHub or Store.
I can't see a reason here for not going into Docker. This knowledge will be more usefull for your team than Vagrant skills.

You can read more about it here: https://stackoverflow.com/questions/16647069/should-i-use-vagrant-or-docker-for-creating-an-isolated-environment


Is there a way to find out every resource that calls a lambda?

if you want to check from where lambda is calling/using/attached you can check it form designer in lambda.

Dashboard of lambda having designer that show lambda is connected with API Gateway , alexa skill etc.

So you can check everything from designer of lambda although you can also use cloud formation stack if you have made it for lambda.

  Here Lambda is connected with cloud watch and amazon EC2

There should be api gateway and other component also


Deploy to server after merge to master branch

One could use conditional builds https://docs.travis-ci.com/user/conditional-builds-stages-jobs/

If code is merged into master one could decide to deploy code to production, but I personally prefer a human intervention by a Product Owner.

jobs:
  include:
      if: branch = master

or

stages:
  - name: deploy
    # require the branch name to be master
    if: branch = master

At other projects I preferred to deploy only tags to enforce the git flow (tags should be created otherwise no deployment to production).

stages:
  - name: deploy
    # require the tag name to match a regular expression
    if: tag =~ ^v1

One could also try (I did not try it yet) whether the if works in combination with script sections:

https://github.com/030/ansible-firefox/blob/master/.travis.yml

Last remark: I would avoid creating such scripts (the ones you included in the question) as most CI tools contain condition checks.


How granular in purpose should FaaS functions be?

In my point of view, you can have a future gain separating the function into multiple parts because it will allow to use another combining system/upload/whatever as input to the processing part and another system to process the images later.

My personal rule of thumb for separation is the main concern:

Step 1 is giving the input, it's already somethhing it itself.
Steps 3 and 4 are, basically, take an input, do the work andgive an output. It waits base64 strings as input and output an image file to a bucket, it should be autonomous in its processing
Step 5 is an api call, it waits image as input and "whatever vision API does" as result/output, that's a task in itself.

I did omit step 2 as it's the trigger of step 3 and 4.

The main interest into this separation of duty is the possibility to work with contracts between the functions (mostly bullet 2 and 3): 2 is expected to output images in a bucket, 3 expects to find images in a bucket, you can then replace 2 or 3 with whatever you want as long as this contract is fulfilled.

Now for when to do this split and is it necessary, if you don't expect things to change in a near future and there's no hard limit approaching, like the time to do the whole work is approaching the max execution time, then there's no need to over anticipate problems in my humble opinion, the whole step 3 to 5 can be seen as a single task unit as you said and this make sense.

My take would be to measure the time within the function to see if there's an interest (for parallel processing for exemple) in taking the combine and transform to image to its own unit.

In brief to answer the question title: there's no silver bullet or hard rule for the granularity, it's always a matter of weighting the benefits toward the cost of slicing the process.


Kubernetes namespaces with RBAC: sufficient for tenant isolation in a public environment?

There's an active SIG devoted to multitenancy so I have a hard time believing that nothing has been written about it in 4 years.
Namespaces are only part of the answer. That isolates kubernetes objects impacting other kubernetes objects, but you also need to worry about shared resources like the ingress namespace. If containers share worker nodes, then there are resource limits to enforce. And the default network model used by kubernetes is flat with DNS exposing services and containers in other namespaces and an open network policy allowing any container to talk to any other container. The kubernetes API is also shared in a multi-tenant environment, allowing a DoS to the kubernetes infrastructure itself. And if users are allowed to run containers with added capabilities, privileged access, host bind mounts, host namespaces, and probably many other features, they can escape the container isolation and get root access on the host running the containers. None of this is trivial to implement securely.


Is it possible to design headless tests with JMeter/Selenium? (and Docker)

JMeter supports headless operation, in fact it's even the recommended mode for best load testing results. From Non-GUI Mode (Command Line mode):

  For load testing, you must run JMeter in this mode (Without the GUI)
  to get the optimal results from it. To do so, use the following
  command options:

  -n
      This specifies JMeter is to run in non-gui mode

As for Selenium, things are a bit debatable, the Does Selenium support headless browser testing? SO post is probably interesting, personally I find Ross Patterson's and General Redneck's answers most useful - basically recommending the use of a virtual X server (Xvfb, for example) instead of a headless browser.

A nice related post: How To Run Your Tests Headlessly with Xvfb.


When validating an input variable of a collection type, we typically want to write a validation rule that tests if something is true for each value within the collection, rather than true of the collection as a whole.
You can write a "for each" sort of condition using Terraform's for expressions. If you are using Terraform v0.14 or later then you can write it concisely using the alltrue function along with a for expression that produces a true or false result for each of the items, like this:
  validation {
    condition = alltrue([
      for id in var.security_groups : can(regex("^sg-", id))
    ])
    error_message = "All security group ids must start with \"sg-\"."
  }

If you are using Terraform v0.13 then you won't have the alltrue function available to you yet, but you can get a similar result in a clunkier way by using the optional if clause of a for expression to filter out all of the valid ids and then checking whether the resulting list is empty -- that is, whether there are any remaining items that were not valid:
  validation {
    condition = length([
      for id in var.security_groups : id
      if !can(regex("^sg-", id))
    ]) == 0
    error_message = "All security group ids must start with \"sg-\"."
  }

This second example is a bit more cryptic to understand because it's using some language features in an atypical way, so I'll break it down into smaller parts to explain:

[for id in var.security_groups : id if !can(regex("^sg-", id))] filters the elements of var.security_groups to keep only the ones where the expression is true.
!can(regex("^sg-", id)) uses the ! operator to turn your original expression that decides if the value is valid into an expression which decides if the value is invalid. Consequently the result of the for expression will be empty unless there's at least one invalid value.
Finally, the length( ... ) == 0 around the for expression says that the value is only valid if filtering out all of the valid values produces an empty list, which conversely means that the list didn't contain any invalid values.


Yes you can create resources (including pods) from a url yaml using kubectl.  However, just because you pointed to a remote url does not mean that the resource will automatically update.  You will need to either re-deploy when the remote yaml is updated, develop/integrate with a service that handles this, or look into using a tool such as helm to help manage changes.

It does not need to be avoided, you just need to know that it will not automatically update if the source changes.


Ansible Expect Module hung on attempting a simulated return keystroke

I figured out the issue. The "Question:" parameter needs to be omitted:
    ---
    - hosts: localhost
      connection: local
      gather_facts: false

  tasks:
    - name: Expect Attempt
      expect:
        echo: yes
        command: telnet 10.233.82.7 2033
        responses:
          Connected to port 33. :  "echo"
          login: "admin"
          Password: "password"


How to set git-lfs as an additional checkout behavior using the JCAC plugin?

It seems that it is only possible by using a workaround as depicted in this answer.

After setting gitLFS in the UI, subsequently inspecting the /var/lib/jenkins/jobs/some-job/config.xml file it became obvious that the XML looked as follows:

<traits>
   <jenkins.plugins.git.traits.GitLFSPullTrait>
      <extension class="hudson.plugins.git.extensions.impl.GitLFSPull"/>
   </jenkins.plugins.git.traits.GitLFSPullTrait>
</traits

After defining jenkins.plugins.git.traits.GitLFSPullTrait:

jobs:
  - script: >
      multibranchPipelineJob("example") {
        branchSources {
          ...
        }
        configure { node ->
          node / sources / data / 'jenkins.branch.BranchSource' / source / traits {
            'jenkins.plugins.git.traits.GitLFSPullTrait'()
          }
        }
      }

Git-lfs was enabled.


I do not see any obstacles. All communication is done through the apiserver over https.
Multiple replicas of the controller and scheduler are allowed, leader election is built in. All you need is load balancing in front of the apiservers.
I find this to work best, since usually the failure domain is a node, rather than a component.
Yes.


Reflect GitLab repo changes in VSTS code

I'm using https://docs.gitlab.com/ee/workflow/repository_mirroring.html#pushing-to-a-remote-repository to push everything into a repository in VSTS, easily done via access token with the right permissions.


I am using lambda in response to objects appearing on S3 buckets.  For me in my particular use case, there are around 600 triggers a day.  I have written our lambda functions to use their stdout as their log file, writing all key events to standard out in a semi-structured textual sentence mixed with key/value pairs for data values, and then feed the cloudwatch logs into splunk.

Here is what I have found...

Lambdas will (occasionally) be triggered multiple times for the same object.
Lambdas will (sometimes) take noticeably longer to run than normal.
To see / diagnose issues caused by concurrent invocations that interfere with each other, the AWS Web Console doesn't do it. You need a tool like splunk.

However, does the value of this work provide enough revenue to cover the costs of using ECS(Elastic Container Service) for your process management?  If you are able to run an ECS container, you could trigger the lambda on S3 object creation to create a message on an SQS message queue. Your ECS container is listening to your queue and when a message arrives, processes it just like within your lambda.  Its output to trigger the next stage is to pop a message onto the SQS queue handling the next task.

This will solve a few problems for you, such as escaping the 5 minute maximum for Lambdas, and creating an easy way to retry failures.

However, it will create a few new issues too.

Creating and deploying docker images is more work than editing lambda on the AWS console.
Infrastructure costs are more fixed in nature, rather than proportional to workload.


A service by AWS that routes traffic based on simple logic

You can use AWS APP MESH to route traffic between OSM and GDM. First, all traffic will be sent to OSM using App Mesh's VirtualRouter.
VirtualRouter is an object to route traffic between K8S services. One service will be pointed to the ECS hosted OSM. Another service will be pointed to GDM using External Service
You can use a Cronjob which will run in a pod and sense timeout-related metrics from ElasticCache. If the condition matched - the code host in the pod and running as a CronJob will modify VirtualRouter.


How to login using serverspec using ssh key?

serverspec is using this Ruby class to ssh to servers

One should set the path to the key into the ~/.ssh/config file:

Host X
    HostName X
    User Y
    PreferredAuthentications publickey
    IdentityFile ~/.ssh/Z

Now rake spec runs the test on the server.


TFS Service hook should not fire when work item is updated via REST API

The service hook will fire as part of the notifications process.  You can suppress all notifications by setting the URI parameter suppressNotifications to true.

An example request URI might be:

PATCH https://dev.azure.com/fabrikam/_apis/wit/workitems/{id}?suppressNotifications=true&api-version=5.0

See the work item update API documentation for more information.


How do I run Ansible Azure playbooks while avoiding storing credentials in files?

Vault password

First of all, you should get familiar with the fact that vault password file can be executable script. In this case Ansible executes it and expects to receive password as its output.

For example you can use gpg-agent or keychain to store your actual password and unlock it when required. Read more in this blog post: https://benincosa.com/?p=3235

If you are a bit paranoid, you can add notification when your password script is called, like this:

#!/bin/bash
PARENT_PROCESS=$(ps -p $PPID -o args | tail -n 1)
osascript -e "display notification \"Vault password used by ${PARENT_PROCESS}\" with title \"Ansible\" sound name \"default\""
gpg --batch --use-agent --no-tty --decrypt key.gpg 2>/dev/null

This vault password script uses key.gpg as actual vault key and also shows popup notification (for MacOS) with parent process name when script is used. Gpg-agent caches unlock password for some time, so there is no need to enter password every time you start playbook.

Just set vault_password_file = ./vault_pass.sh in your ansible.cfg.

Environment

You said that you use azure_rm.py as dynamic inventory script. This means that you have to set credentials into your environment variables before you start ansible-playbook for it to be able to use them.

You can make two files:

secure_env (encrypted with vault):

export AZURE_SECRET=xxx;export AZURE_SUBSCRIPTION_ID=xxx;

set_env (plain text):

echo -n "Setting secure vars... "
eval $(ansible-vault view secure_env)
echo "done."

When you open new terminal to execute your automation tasks, you have to run:

source set_env

At this moment, bash evaluates set_env and secure_env (decrypted via ansible-vault). After this command you have Azure credentials defined for the current shell, so you can execute playbooks as usual:

ansible-playbook provision-my-azure-instances.yml

So using this two approaches, you can store key.gpg and secure_env in your repository; then in the new terminal call source set_env once, enter gpg password once (to unlock future use of key.gpg); then call ansible-playbook as many times as you like without any passwords.


After removing Docker container, why is it automatically re-created?

Check the following:

Something like watchtower maybe running and monitoring your host machine on which the containers are running. Watchtower can re-pull your images from a registry and re-launch a container for you. There are some tools similar to watchtower which do this.


Does a company need DevOps if it already delivers software at acceptable speed?

DevOps isn't just a different way of doing things, it is a better way of doing things. Perhaps it will help me to make my point if I rephrase the question in a different context - the context of the industrial age, the railroad and the steam hammer:

Does a railroad company need Steam Hammers if it already lays track at an acceptable speed?

I mean, steam hammers from the business view are about speed and quality, right?
What does it mean for a construction company contracting for the railroad that is laying track "fast enough" for their customers without steam hammers? Which of these might apply?

They lay track at a sufficient level, no change is required

They use steam hammers but don't know it or call the steam hammer something else

They use something totally different and better than a steam hammer and we really should find out what it is (before it is too late)

They are awfully wrong: their speed and quality are below their knowledge and they will know it when they begin to lose bids to other contractors.

The answer is obviously probably #4. This hypothetical company may not see the problem with the next financial report - perhaps it will take a few years - but eventually it will become clear how and why competitors outperform businesses still manually "laying track" as it were. Sure, it's possible they have come up with a diesel powered hammer that is more efficient and less prone to malfunction or breakdowns, but not likely.
Thinking that they can outperform the steam hammer is going to get this company left behind. Sure, an individual group may have their steam hammer blow up and fall apart and perhaps that one group has such trouble with their steam hammer that it would have been more efficient in that one case to do things the old fashioned way, but they are the exception to the rule and the problem likely was that their steam hammer was poorly built and/or poorly maintained.
The industrial age is coming and the railroad company can either check out the competition and then try to get their own steam hammers, or get left behind and eventually succumb to their irrelevance and antiquity.
DevOps is no different. It's just another iteration of automation that companies can either embrace or ignore at their own peril much like the steam hammer.


Why is Prometheus not a good choice for data with high cardinality?

As far as I know, Prometheus doesn't mind high-cardinality data. What Prometheus doesn't like is high-cardinality labels.

Let's start with Prometheus official documentation, it gives a good high-level explanation why:

  CAUTION: Remember that every unique combination of key-value label
  pairs represents a new time series, which can dramatically increase
  the amount of data stored. Do not use labels to store dimensions with
  high cardinality (many different label values), such as user IDs,
  email addresses, or other unbounded sets of values.

https://prometheus.io/docs/practices/naming/

The important part is really that unique combinations of key-value creates new time series in Prometheus.

For instance, if you store a gauge-type metric registration_complete that records the time a user took to complete his/her registration form, Prometheus won't have any issue with that. You'll have one time series that have hundred of thousand of values: one value for each user that registered (how long it took). You'll be able to graph that metric over time, get the p95, etc.

If you want to add some cardinality to it, you could add a label like region: US-east, Asia-Pacific, etc. You'll be able to graph all regions and compare them or maybe group them all. The number of regions is probably low (<10) and it is certainly bound. If on AWS, there's a fixed number of regions and they don't change much over time. Sure, AWS might add, remove or rename a region, but that's not changing by the minute and there's not thousands of them.

So to come back to what Prometheus advise against: you should not create high-cardinality labels. You should not add a user_id label to your registration_complete metric. If you do so, you'll have hundred of thousands of different time series, one for each user!, and they will all have only one data point. That's really the worst case scenario.

In this case, in order to graph the registration_complete metric over all labels, Prometheus will have to query all the individual time series (the X thousands of them) and aggregate all of them.

You said you come from a SQL background, so I'll attempt an analogy. Unique combination of key-value label pairs creating new time series in Prometheus amounts to individual tables in SQL. Having a label user_id equals to having one table per user_id.

Note: not all TSDB work the same way and I can't speak for them all.


Ansible execute task on different hosts in order

Q: "Is it possible to use Ansible to manage the order of programs?"
A: Yes. It's possible. For example, the playbook below does the job
- hosts: HostA
  tasks:
    - include_role:
        name: startprogram

- hosts: HostB
  tasks:
    - include_role:
        name: runcommand
      when: hostvars.HostA.HostAprogram_started|default(false)

- hosts: HostC
  tasks:
    - include_role:
        name: startsecprogram
      when: hostvars.HostB.HostBcommand_passed|default(false)

- hosts: HostD,HostE
  tasks:
    - debug:
        msg: command on {{ inventory_hostname }} which requires HostC program to run
      when: hostvars.HostC.HostCprogram_started|default(false)

Example of the roles for the purpose of testing
shell> cat roles/startprogram/tasks/main.yml
- debug:
    msg: /etc/init.d/HostAprogram.sh start
- set_fact:
    HostAprogram_started: true

shell> cat roles/runcommand/tasks/main.yml
- debug:
    msg: command which using host A
- set_fact:
    HostBcommand_passed: true

shell> cat roles/startsecprogram/tasks/main.yml
- debug:
    msg: /etc/init.d/HostCprogram.sh start
- set_fact:
    HostCprogram_started: true

Example of the output
ok: [HostA] => {
    "msg": "/etc/init.d/HostAprogram.sh start"
}
ok: [HostB] => {
    "msg": "command which using host A"
}
ok: [HostC] => {
    "msg": "/etc/init.d/HostCprogram.sh start"
}
ok: [HostD] => {
    "msg": "command on HostD which requires HostC program to run"
}
ok: [HostE] => {
    "msg": "command on HostE which requires HostC program to run"
}


Autorun ElasticSearch with Docker Compose

https://github.com/elastic/stack-docker/blob/master/docker-compose.yml

This is an official docker-compose for running Elasticsearch, and will make a good starting point for whatever you're trying to achieve. This would be the key part:

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:${TAG}
    environment: ['http.host=0.0.0.0', 'transport.host=127.0.0.1', 'ELASTIC_PASSWORD=${ELASTIC_PASSWORD}']
    ports: ['127.0.0.1:9200:9200']
    networks: ['stack']

But I'd recommend cloning the entire repository locally and running docker-compose up to experiment.


What do you tell someone new to going oncall or PagerDutyl?

There are a number of things that I think it's important to know before going on-call:

How to triage an issue.  There should be a defined list of priorities with examples so that you can identify how serious a problem is.
What the priority levels mean.  Can you ack this and then go back to sleep?
Who to contact post-triage. The on-caller's primary responsibility is triage, not fixing.  Fixing the problem is good, too, but once you've triaged the issue, then it's up to you to rope in other folks from other teams and manage communications between them.

People jumping straight into fixing problems before triaging is the most common mistake I've seen (because, after all, we're used to fixing things and think of that as our job).  But communications are often vital, and doing things in the wrong order can mean customers are left in the dark because you got caught up in trying to fix a problem before letting your support people know there is a problem.


So you can use two build stages in your Dockerfile so you don't end up having unnecessary things inside container.  Anything from first stage is gone until you move it to second stage. Something like:

FROM whatever as build-stage
WORKDIR /app
RUN apt-get update && apt-get install -y unzip
RUN wget https://getcomposer.org/download/1.8.3/composer.phar
RUN php installer.php

FROM whatever as production-stage
COPY --from=build-stage /app ./


Persistent Storage with Docker in Production - Which Solution and Why?

I can answer the 2nd point:

Docker is most suited in a micro service based architecture when the application runs inside the containers but the storage or any other live sessions are maintained in shared RAM or the database.

Basically you just should not store anything inside the docker container. There are many reasons to it:

Consider upgrade: Someone from your team has created a newer image of the application and you need the container running with the latest image. The current docker and popular way of doing this is to bring the existing container down and spin a new container with the same run time parameters as the older container but with the newer image. This is one of the biggest reasons why containers should always be stateless and not contain any data. You can have all your data mounted to some place and sessions kept inside a db or something like memchached etc.
One of the big use case of docker is to build clusters. If you start keeping data inside your containers then its an overhead to maintain that data in sync between the application containers.
The docker community in general does not recommend to keep any data in container and hence nobody has tried taking this risk in production and nobody wants to be the first storyteller of how they messed up production :)


How to make Octopus Deploy trigger a new release when a new release made on GitHub?

Octopus can be configured as either the target of a push from GitHub or you can set up a feed in Github, and pull from that feed.
In your case, it sounds like you want a push mechanism, such as GitHub Actions to do the heavy lifting.


Load can mean a lot with an ELK stack.

For ingestion load, the logstash parsing side, you're looking at a combination of one or two things:

CPU load on the ingestion boxes.
If used, back-pressure in your ingestion-queue (such as Redis, AMQP, or others)

This can be simple stuff like queue-size, or a fancy number like age of the oldest member of the queue.

If you're in a cloud-provider that gives you a way to auto-scale your ingestion nodes, keying on CPU functions pretty well in recent Logstash versions.

For ingestion and searching, the ElasticSearch side, it can depend on a couple of things due to your architecture. If you're using physical servers, it is very easy to have far more CPU capacity than I/O capacity to store the processed results. If you're using physical servers, keep an eye on I/O queue depth in addition to CPU loading. Also beware of the Compressed Object Pointer barrier, around 32GB of JVM HEAP. Stay under 32GB. Just do it.

If you're on cloudy instances, its harder to out-strip your I/O due to the reduced core-counts. Even so, queue-depth and cpu-loading are good things to keep an eye on. If your JVM monitoring tools are up to it, tracking Garbage Collection frequency is worth it until you have a feel for what you need for RAM.

For all of this, keeping an eye on consumed disk-space will tell you when you need to add more ES data-nodes or adjust your index retention settings.

For display and searching, such as with Kibana or Grafana, I've found this to be the part of the stack I never bother looking at. At least in any way other than up/down tracking. We track CPU, but we don't alarm anything since in our experience even a 2 core box with 7GB of RAM is enough to keep up without breaking a sweat. Both of these work OK with horizontal scaling if you do run into problems.

ELK stacks come in many sizes, from all in one boxes, all the way up to systems with routing logic to bring events to parsing and ES clusters owned by completely different parts of the organization. The monitoring needs of a small architecture (all in one) are rather different than medium architectures (discrete parsing tier, usually queue-mediated), and are different again from large architectures (multiple parsing-tiers with multiple event-stores).


ingrex-nginx in kubernetes with express node.js does not return req.currentUser

Try using this:- use-forwarded-headers
apiVersion: v1
kind: ConfigMap
data:
  use-forwarded-headers: "true"
metadata:
  name: nginx-configuration
  namespace: default


Change Ansible module based on variable or group

Elaborating on @030's answer, here's how you can use variables for modules:

You need to group the target hosts in groups, for the example I'm using server1.example.com as member of hosts1 group and server2.example.com as member of hosts2 group.
You create group variables:

group_vars/hosts1
group_vars/hosts2

Both files contain a variable named module with different value, in the test I use stat.

Your playbook will look like this:

test.yml

---
 - hosts: all
   user: me
   tasks:
     - name: test module variables
       action: "{{ module }} path=/home/me/myfile"

The result of the play:

[me@myserver]$ ansible-playbook test.yml -i test

PLAY [all] ********************************************************************************

TASK [Gathering Facts] ********************************************************************
ok: [server1.example.com]
ok: [server2.example.com]

TASK [test module variables] **************************************************************
ok: [server1.example.com]
ok: [server2.example.com]

PLAY RECAP ********************************************************************************
server1.example.com        : ok=2    changed=0    unreachable=0    failed=0
server2.example.com        : ok=2    changed=0    unreachable=0    failed=0


403 forbidden error on calling jenkins post api from python script

As recommended by Jenkins (>2.2xx) you may benefit by creating an API token for the user, I did so, to trigger Jenkins job in my local Jenkins server from NODEJS code.
Please refer below for further clarification:

https://www.jenkins.io/doc/book/system-administration/authenticating-scripted-clients/#shell-with-curl
https://www.jenkins.io/doc/book/using/remote-access-api/#RemoteaccessAPI-CSRFProtection

And for details: https://www.jenkins.io/doc/book/managing/security/


The fact that all the code is in a single repository doesn't mean that all the code is changing every time a commit is pushed. I would first make the "pathways" in the code explicit. E.g. perhaps you have a few subdirectories:

- App1/
  - Docs/
  - Code/
  - Tests/
  - Makefile
- App2/
  - Docs/
  - Code/
  - Tests/
  - Makefile

Assuming that App1 is independent from App2, you could trigger it's build by checking if anything in that directory changed (see https://stackoverflow.com/questions/424071/how-to-list-all-the-files-in-a-commit).

Most continuous integration tools have a git plugin or native feature that allows you to filter on a path. If you were using Jenkins, you could set up a few jobs, each with the included region to be be the pathway that should be triggered -- so, e.g. changes to App1/* would trigger jobs/app1/ which executes make -f App1/Makefile or similar.

The trick then to maintaining this project is really in the Makefiles, or other build tool of your choice. You would write the dependencies in there.

You could also have a single global Makefile and use the git diff-tree trick in the question mentioned above:

if grep -q App1 `git diff-tree --no-commit-id --name-only -r <commit>` ; then
  # trigger pipeline App1
  cd App1
  make build
  make test
  make publish
fi


How to automate submission to Amazon Marketplace?

Snapshot Permissions

Boto3 has a function that allows you to create volume permissions, which is what AMI Sharing with AWS Marketplace requires you to do. snapshot.modify_attribute will allow you to share your AMI with the marketplace account like so (you can also use a JSON representation if you prefer, it's in the docs):

response = snapshot.modify_attribute(
    Attribute = 'createVolumePermission',
    OperationType = 'add',
    UserIds = [
        '679593333241', # Marketplace user ID
    ]
)

There is also an official API function that will help you achieve the permissions requirement listed in AMI Sharing with AWS Marketplace if you decide to migrate from boto3 (or if a reader isn't using boto3). If your AMI is private, you need only make this API call:

https://ec2.amazonaws.com/?Action=ModifySnapshotAttribute
&SnapshotId=ID_HERE
&CreateVolumePermission.Add.1.UserId=679593333241
&AUTHPARAMS

This should add the permission that the marketplace needs to access your AMI. As for replacing ID_HERE with the snapshot's ID, you can do that with boto3 by using your snapshot object's snapshot_id property snapshot.snapshot_id before making the API call (that being said, there's little reason for you to use this API since you have to use boto3 anyways).

Product Metadata

You should be able to use the relevant ServiceCatalog functions in boto3 such as create_product to upload product metadata. There are also equivalent official API functions (this one is just CreateProduct, they're not listed in a way that allows me to link to them nicely). It's worth noting that the official API only accepts JSON whereas boto3 will generate the JSON itself. Which one is more useful to you depends on how your data is structured in your product metadata file(s). Since I don't know that I can't directly write the code to do it.


How to configure Docker tooling on Eclipse?

Under Windows 10, you'll find the Docker Machine (docker-machine.exe) under: C:\Program Files\Docker\Docker\resources\bin

This is the value you need to put in the Docker Machine field.
You don't need to change anything for the VM Driver field as VirtualBox doesn't need to be installed when you use Docker for Windows.

Then you have to edit some setting of the Docker daemon to allow your Eclipse Docker plugin to connect to it:

First of all, check if the Docker daemon is running by clicking on the Task View button on the Windows taskbar.
Mouse over the Docker icon and the tooltip should say: Docker is running.
If it's running, right-click the Docker icon to open the Docker contextual menu and go to Settings.
In the General Tab, which you will land on by default, Make sure the
"Expose daemon on tcp://localhost:2375 without TLS" checkbox is checked.
(It isn't by default)

You can now edit your Docker Connection in Eclipse and do not need to search for a connection name, simply enter how you want to name it (ie. SomeDockerConnection) and check the "TCP Connection" radio button.

You can enter tcp://localhost:2375 in the URI and click "Test Connection".
You should see a message appear to confirm that the ping was successful.

At this point you'll be able to see your containers and images in Eclipse's Docker Explorer perspective.


Pass parameters to jenkins job with the help of userContent

Looks like I read documentation not attentively.

userContent supports only static content but not dynamic.

Thus, PHP/JS or something similar can not be used there: https://wiki.jenkins.io/display/JENKINS/User+Content


You need to add a host entry of mylocalserver in the host file against 127.0.0.1, like 127.0.0.1 mylocalserver.
For /consul, there should be a /consul endpoint in your code to serve your request.

As per your configuration, your http://localhost will be served because you have set proxypass for "/"(all). It will send all requests to the application/server and will be served.


You can set a maximum duration of up to 12 hours in the IAM Role config:

Hope that helps.


If the OS has a python installation, you can use  Ansible to manage it.. Note that you should check the individual module requirements (for example the snap module) since they may have dependencies which cannot be satisfied by your OS.

In general though the modules are written in python and as such a python runtime is needed.


Proper way to showcase multi-container web app to customers

You mention that you are demonstrating this for a customer. The first question you should ask yourself (or, more pertinently, the customer) is :

  what does this customer really want?

Why have they come to you for this work? What is the pain they are currently feeling, what value are you providing to them.

Then, compare and contrast - show them the delta that your solution brings to them.

It may be for example that they want to maintain a legacy application, but upgrade the hardware or underlying components ? In that case, show them how you have contained them with no negative experience to their users.
It may be that they need to scale the capacity for usage of their application   - they want to be able to meet demand elastically. Show them how by deploying in a container setup, you can scale out.
It may be that they need four nines reliability - show them that you can have an easier HA setup - knock out one of the instances and prove that the application still works.
It may be that they want faster delivery with no little impact on quality. Show that your delivery pipeline works smoothly with some automated tests and QA, with delivery and deployment.

TL;DR : Find out what your customer wants. Demonstrate the existing way of doing things. Compare it with the "new" way, and highlight the delta.


Deploy two independently hosted services in production at once without any downtime

I think you're taking the problem on the wrong side, you may start a whole new infrastructure aside and then switch loadbalancers from blue to green, loosing sessions, breaking ongoing transactions etc.

The proper way to achieve what you're doing is coding differently, when A relasease a breaking API ni version n, a should be able to continue to answer call to api in version n-1. the usual method is to have the version in the uri or an header, for exemple:

<a-host>/api/1/getCustomers and <a-host>/api/2/getCustomerList (for exemple if n is 2). With that, you can do a rolling update of B, with backends still using api v1 and new ones using api v2.


The task which fails is:

- name: copy demo app source
  copy: src=demo/app/ dest=/var/www/demo mode=0755
  notify: restart apache2

If you have a look at your folder structure and more specifically the demo_app role, you have:

│   ├── demo_app
│   │   ├── defaults
│   │   │   └── main.yml
│   │   ├── files
│   │   │   └── app
│   │   │       ├── demo.py
│   │   │       └── requirements.txt
│   │   ├── handlers
│   │   │   └── main.yml
│   │   ├── meta
│   │   │   └── main.yml
│   │   ├── README.md
│   │   ├── tasks
│   │   │   └── main.yml
│   │   ├── templates
│   │   │   └── demo.wsgi.j2
│   │   ├── tests
│   │   │   ├── inventory
│   │   │   └── test.yml
│   │   └── vars
│   │       └── main.yml

The copy task will look for the src in demo_app/files/. So, either change your task to:

- name: copy demo app source
  copy: src=app dest=/var/www/demo mode=0755
  notify: restart apache2

Or create a demo directory in demo_app/files/ and place the app directory in it (like demo_app/files/demo/app/).

Setting an absolute path to the directory in the task will also work, but don't think it's an elegant solution.


M2 Repo contents is not available in artifactory server

Welcome, Vivek

This is not how it works

Your local cache (~/.m2) is not transferred to Artifactory if anything the direction is the other way around (in some point of time you'll see files from Artifactory appear in your local cache).

When you run a Maven build, the configuration you provided in Maven (check the Set Me Up button in Artifactory UI) will make Maven searching for dependencies in the following order:

Local cache (~./m2)
Artifactory (first local, then remote caches)
If Artifactory won't find the artifact, whenever willows-remote is pointing to.

After the artifact is found, it will be saved in Artifactory cache and in a local cache on your machine.

The <distributionManagement> tag you used in your pom.xml is for deploying the artifacts you created when you run a mvn deploy command. It will take the artifact produced (one file, not your entire ~/.m2 cache) and upload it to the repo specified in <distributionManagement>.


AWS API Gateway: 403 forbidden with disabled API key; error from cloudfront

When you deploy an edge-optimized API, API Gateway sets up an Amazon CloudFront distribution and a DNS record to map the API domain name to the CloudFront distribution domain name. Requests for the API are then routed to API Gateway through the mapped CloudFront distribution.

Source: API Gateway documentation — Edge-optimized custom domain names.

To allow calls to a method of a resource in your API without API key, set its API Key Required setting to false:


Jenkins and Kubernetes CLI plugin and GKE

Put it inside the sh?
withKubeConfig(...) {
  sh “””
    export ...
    kubectl ...
  “””
}